External quality assessments for SARS-CoV-2 genome detection in Austria : A comparison of the first postpandemic round with results from the pandemic era.

BACKGROUND: External quality assessment (EQA) schemes provide objective feedback to participating laboratories about the performance of their analytical systems and information about overall regional analytical performance. The EQAs are particularly important during pandemics as they also assess the reliability of individual test results and show opportunities to improve test strategies. With the end of the COVID-19 pandemic, the testing frequency significantly decreased in Austria. Here, we analyzed whether this decrease had an effect on participation and/or performance in SARS-CoV­2 virus detection EQAs, as compared to the pandemic era. MATERIAL AND METHODS: Identical samples were sent to all participating laboratories, and the EQA provider evaluated the agreement of the reported results with defined targets. The EQA was operated under two schemes with identical samples and therefore we analyzed it as a single EQA round. The performance of testing was reported as true positive ratios, comparing the post-pandemic data to previous rounds. Furthermore, subgroups of participants were analyzed stratified by laboratory type (medical or nonmedical) and the test system format (fully automated or requiring manual steps). RESULTS: While the frequency of false negative results per sample did not change during the 3 years of the pandemic (5.7%, 95% confidence interval [CI] 3.1-8.4%), an average per sample false negative ratio of 4.3% was observed in the first post-pandemic EQA (0%, 1.8%, and 11% for the 3 positive samples included in the test panel, n = 109 test results per sample). In this first post-pandemic EQA medical laboratories (average 0.4% false negative across 3 samples, n = 90) and automated test systems (average 1.2% false negative, n = 261) had lower false negative ratios than nonmedical laboratories (22.8%, n = 19) and manual test systems (16.7%, n = 22). These lower average ratios were due to a low concentration sample, where nonmedical laboratories reported 36.8% and manual test systems 54.5% true positive results. CONCLUSION: Overall ratios of true positive results were below the mean of all results during the pandemic but were similar to the first round of the pandemic. A lower post-pandemic true positive ratio was associated with specific laboratory types and assay formats, particularly for samples with low concentration. The EQAs will continue to monitor the laboratory performance to ensure the same quality of epidemiological data after the pandemic, even if vigilance has decreased.

Performance of SARS-CoV-2 nucleic acid amplification testing in Austria as measured by external quality assessment schemes during 3 years of the COVID-19 pandemic: an observational retrospective study.

BACKGROUND: The aim of external quality assessment (EQA) schemes is to evaluate the analytical performance of laboratories and test systems in a near-to-real-life setting. This monitoring service provides feedback to participant laboratories and serves as a control measure for the epidemiological assessment of the regional incidence of a pathogen, particularly during epidemics. Using data from EQA schemes implemented as a result of the intensive effort to monitor SARS-CoV-2 infections in Austria, we aimed to identify factors that explained the variation in laboratory performance for SARS-CoV-2 detection over the course of the COVID-19 pandemic. METHODS: For this observational study, we retrospectively analysed 6308 reverse transcriptase quantitative PCR (RT-qPCR) test results reported by 191 laboratories on 71 samples during 14 rounds of three SARS-CoV-2 pathogen detection EQA schemes in Austria between May 18, 2020, and Feb 20, 2023. We calculated the overall rates of false and true-negative, false and true-positive, and inconclusive results. We then assessed laboratory performance by estimating the sensitivity by testing whether significant variation in the odds of obtaining a true-positive result could be explained by virus concentration, laboratory type, or assay format. We also assessed whether laboratory performance changed over time. FINDINGS: 4371 (93·7%) of 4663 qPCR test results were true-positive, 241 (5·2%) were false-negative, and 51 (1·1%) were inconclusive. The mean per-sample sensitivity was 99·7% in samples with high virus concentrations (1383 [99·4%] true-positive, three [0·2%] false-negative, and five [0·4%] inconclusive results for 1391 tests in which the sample cycle threshold was ≤32), whereas detection rates were lower in samples with low virus concentrations (mean per-sample sensitivity 92·5%; 2988 [91·3%] true-positive, 238 [7·3%] false-negative, and 46 [1·4%] inconclusive results for 3272 tests in which the cycle threshold was >32). Of the 1645 results expected to be negative, 1561 (94·9%) were correctly reported as negative, 10 (0·6%) were incorrectly reported as positive, and 74 (4·5%) were reported as inconclusive. Notably, the overall performance of the tests did not change significantly over time. The odds of reporting a correct result were 2·94 (95% CI 1·75-4·96) times higher for a medical laboratory than for a non-medical laboratory, and 4·60 (2·91-7·41) times greater for automated test systems than for manual test systems. Automated test systems within medical laboratories had the highest sensitivity when compared with systems requiring manual intervention in both medical and non-medical laboratories. INTERPRETATION: High rates of false-negativity in all PCR analyses evaluated in comprehensive, multiple, and repeated EQA schemes outline a clear path for improvement in the future. The performance of some laboratories (eg, non-medical laboratories or those using non-automated test systems) should receive additional scrutiny-for example, by requiring additional EQA schemes for certification or accreditation-if the aggregated data from EQA rounds suggest lower sensitivity than that recorded by others. This strategy will provide assurances that epidemiological data as a whole are reliable when testing on such a large scale. Although performance did not improve over time, we cannot exclude extenuating circumstances-such as shortages and weakened supply chains-that could have prevented laboratories from seeking alternative methods to improve performance. FUNDING: None.

Design of external quality assessment schemes and definition of the roles of their providers in future epidemics.

During an epidemic, individual test results form the basis of epidemiological indicators such as case numbers or incidence. Therefore, the accuracy of measures derived from these indicators depends on the reliability of individual results. In the COVID-19 pandemic, monitoring and evaluating the performance of the unprecedented number of testing facilities in operation, and novel testing systems in use, was urgently needed. External quality assessment (EQA) schemes are unique sources of data reporting on testing performance, and their providers are recognised contacts and support for test facilities (for technical-analytical topics) and health authorities (for planning the monitoring of infection diagnostics). To identify information provided by SARS-CoV-2 genome detection EQA schemes that is relevant for public health microbiology, we reviewed the current literature published in PubMed between January, 2020, and July, 2022. We derived recommendations for EQA providers and their schemes for best practices to monitor pathogen-detection performance in future epidemics. We also showed laboratories, test facilities, and health authorities the information and benefits they can derive from EQA data, and from the non-EQA services of their providers.

Three rounds of a national external quality assessment reveal a link between disharmonic anti-SARS-CoV-2 antibody quantifications and the infection stage.

OBJECTIVES: The WHO's standardized measuring unit, "binding antibody units per milliliter (BAU/mL)," should allow the harmonization of quantitative results by different commercial Anti-SARS-CoV-2 immunoassays. However, multiple studies demonstrate inter-assay discrepancies. The antigenic changes of the Omicron variant affect the performance of Spike-specific immunoassays. This study evaluated the variation of quantitative Anti-SARS-CoV-2-Spike antibody measurements among 46, 50, and 44 laboratories in three rounds of a national external quality assessment (EQA) prior to and after the emergence of the Omicron variant in a diagnostic near-to-real-life setting. METHODS: We analyzed results reported by the EQA participant laboratories from single and sequential samples from SARS-CoV-2 convalescent, acutely infected, and vaccinated individuals, including samples obtained after primary and breakthrough infections with the Omicron variant. RESULTS: The three immunoassays most commonly used by the participants displayed a low intra-assay and inter-laboratory variation with excellent reproducibility using identical samples sent to the participants in duplicates. In contrast, the inter-assay variation was very high with all samples. Notably, the ratios of BAU/mL levels quantified by different immunoassays were not equal among all samples but differed between vaccination, past, and acute infection, including primary infection with the Omicron variant. The antibody kinetics measured in vaccinated individuals strongly depended on the applied immunoassay. CONCLUSIONS: Measured BAU/mL levels are only inter-changeable among different laboratories when the same assay was used for their assessment. Highly variable ratios of BAU/mL quantifications among different immunoassays and infection stages argue against the usage of universal inter-assay conversion factors.

Converting to an international unit system improves harmonization of results for SARS-CoV-2 quantification: Results from multiple external quality assessments.

BACKGROUND: The detection of SARS-CoV-2 vRNA in clinical samples has relied almost exclusively on RT-qPCR as the gold standard test. Published results from various external quality assessments ("ring trials") worldwide have shown that there is still a large variability in results reported for the same samples. As reference standards of SARS-CoV-2 RNA are available, we tested whether using standard curves to convert Ct values into copies/mL (cp/mL) improved harmonization. METHODS: Nine laboratories using 23 test systems (15 of which were unique) prepared standard dilution curves to convert Ct values of 13 SARS-CoV-2 positive samples to cp/mL (hereafter IU/mL). The samples were provided in three rounds of a virus genome detection external quality assessment (EQA) scheme. We tested the precision and accuracy of results reported in IU/mL, and attempted to identify the sources of variability. RESULTS: Reporting results as IU/mL improved the precision of the estimated concentrations of all samples compared to reporting Ct values, although some inaccuracy remained. Variance analysis showed that nearly all variability in data was explained by individual test systems within individual laboratories. When controlling for this effect, there was no significant difference between all other factors tested (test systems, EQA rounds, sample material). CONCLUSIONS: Converting results to copies/mL improved precision across laboratory test systems. However, it seems the results are still very specific to test systems within laboratories. Further efforts could be made to improve accuracy and achieve full harmonization across diagnostic laboratories.

Results of a SARS-CoV-2 virus genome detection external quality assessment round focusing on sensitivity of assays and pooling of samples.

OBJECTIVES: Results of earlier external quality assessment (EQA) rounds suggested remarkable differences in the sensitivity of SARS-CoV PCR assays. Although the test systems are intended to detect SARS-CoV-2 in individual samples, screening is often applied to sample pools to increase efficiency and decrease costs. However, it is unknown to what extent these tests actually meet the manufacturer's specifications for sensitivity and how they perform when testing sample pools. METHODS: The sensitivity of assays in routine use was evaluated with a panel of positive samples in a round of a SARS-CoV-2 virus genome detection EQA scheme. The panel consisted of samples at or near the lower limit of detection ("weakly positive"). Laboratories that routinely test sample pools were asked to also analyze the pooled EQA samples according to their usual pool size and dilution method. RESULTS: All participants could detect a highly positive patient-derived sample (>106 copies/mL). Most (96%) of the test systems could detect at least 1,000 copies/mL, meeting the minimum acceptable benchmark, and many (94%) detected the vRNA in a sample with lower concentration (500 copies/mL). The false negative ratio increased to 16 and 26% for samples with 100 and 50 copies/mL, respectively. CONCLUSIONS: The performance of most assays met or exceeded their specification on sensitivity. If assays are to be used to analyze sample pools, the sensitivity of the assay and the number of pooled samples must be balanced.

External quality assessment providers' services appear to more impact the immunohaematology performance of laboratories than national regulatory and economic conditions.

OBJECTIVES: Medical laboratories may, at their own discretion, exceed but not undercut regulatory quality requirements. Available economic resources, however, may drive or hinder eagerness to exceed minimum requirements. Depending on the respective scopes of regulatory and economic framework conditions, differing levels of quality efforts to safeguard laboratory performance can be anticipated. However, this has not yet been investigated. METHODS: Immunohaematology external quality assessment (EQA) results collected by 26 EQA providers from their participant laboratories in 73 countries from 2004 to 2019 were evaluated. Error rates were aggregated in groups according to the respective national regulatory and economic framework conditions, to whether or not expert advice was provided in case of incorrect results, and the frequency of EQA samples. RESULTS: These representative data indicate no association between national regulatory (mandatory participation in EQA, monitoring of performance of individual laboratories by authorities, financial consequences of incorrect results) and economic (level of national income, share of national health expenditure) conditions to the quality performance of medical laboratories in immunohaematology. However, EQA providers' support for laboratories in the event of incorrect results appear to be associated with lower error rates, but a high EQA sample frequency with higher error rates. CONCLUSIONS: Further research into the impact of introducing or changing services of EQA providers is needed to confirm the results found in this first of its kind study.

Inadequate design of mutation detection panels prevents interpretation of variants of concern: results of an external quality assessment for SARS-CoV-2 variant detection.

OBJECTIVES: Mutation-specific PCR assays have quickly found their way into laboratory diagnostics due to their capacity to be a fast, easy to implement and high-throughput method for the detection of known SARS-CoV-2 variants of concern (VoCs). However, little is known about the performance of such assays in routine laboratory analysis. METHODS: The results reported in a recent round of an external quality assessment (EQA) scheme for SARS-CoV-2 mutation-specific PCR were retrospectively analyzed. For the determination of individual variant-specific sequences as well as for the interpretation results for certain virus variants, correct, incorrect, and unreported results were evaluated, and their possible causes were investigated. RESULTS: A total of 34 laboratories participated in this study. For five samples containing the VoC Alpha + E484K, Beta, Gamma, Delta, or B.1.1.318 (as a variant of interest), 848 results for SARS-2-CoV mutation detection were reported, 824 (97.2%, range per sample 88-100%) of which were correct. Melting curve assays gave 99% correct results, real-time RT-qPCR 94%, microarray-based assays 100%, and MALDI-TOF MS 96%. A total of 122/167 (73%) reported results for SARS-CoV-2 variant determination were correct. Of the 45 inconclusive or incorrect results, 33 (73%) were due to inadequate selection of targets that did not allow identification of contemporary VoC, 11 (24%) were due to incorrect results, and one (3%) was due to correct results of mutation-specific PCR. CONCLUSIONS: Careful and up-to-date selection of the targets used in mutation-specific PCR is essential for successful detection of current SARS-CoV-2 variants.

The versatility of external quality assessment for the surveillance of laboratory and in vitro diagnostic performance: SARS-CoV-2 viral genome detection in Austria.

OBJECTIVES: External quality assessment (EQA) schemes provide information on individual and general analytical performance of participating laboratories and test systems. The aim of this study was to investigate the use and performance of SARS-CoV-2 virus genome detection systems in Austrian laboratories and their preparedness to face challenges associated with the pandemic. METHODS: Seven samples were selected to evaluate performance and estimate variability of reported results. Notably, a dilution series was included in the panel as a measure of reproducibility and sensitivity. Several performance criteria were evaluated for individual participants as well as in the cohort of all participants. RESULTS: A total of 109 laboratories participated and used 134 platforms, including 67 different combinations of extraction and PCR platforms and corresponding reagents. There were no false positives and 10 (1.2%) false negative results, including nine in the weakly positive sample (Ct ∼35.9, ∼640 copies/mL). Twenty (22%) laboratories reported results of mutation detection. Twenty-five (19%) test systems included amplification of human RNA as evidence of proper sampling. The overall linearity of Ct values from individual test systems for the dilution series was good, but inter-assay variability was high. Both operator-related and systematic failures appear to have caused incorrect results. CONCLUSIONS: Beyond providing certification for participating laboratories, EQA provides the opportunity for participants to evaluate their performance against others so that they may improve operating procedures and test systems. Well-selected EQA samples offer additional inferences to be made about assay sensitivity and reproducibility, which have practical applications.

Variability of cycle threshold values in an external quality assessment scheme for detection of the SARS-CoV-2 virus genome by RT-PCR.

OBJECTIVES: The qualitative results of SARS-CoV-2 specific real-time reverse transcription (RT) PCR are used for initial diagnosis and follow-up of Covid-19 patients and asymptomatic virus carriers. However, clinical decision-making and health management policies often are based additionally on cycle threshold (Ct) values (i.e., quantitative results) to guide patient care, segregation and discharge management of individuals testing positive. Therefore, an analysis of inter-protocol variability is needed to assess the comparability of the quantitative results. METHODS: Ct values reported in a SARS-CoV-2 virus genome detection external quality assessment challenge were analyzed. Three positive and two negative samples were distributed to participating test laboratories. Qualitative results (positive/negative) and quantitative results (Ct values) were assessed. RESULTS: A total of 66 laboratories participated, contributing results from 101 distinct test systems and reporting Ct values for a total of 92 different protocols. In all three positive samples, the means of the Ct values for the E-, N-, S-, RdRp-, and ORF1ab-genes varied by less than two cycles. However, 7.7% of reported results deviated by more than ±4.0 (maximum 18.0) cycles from the respective individual means. These larger deviations appear to be systematic errors. CONCLUSIONS: In an attempt to use PCR diagnostics beyond the identification of infected individuals, laboratories are frequently requested to report Ct values along with a qualitative result. This study highlights the limitations of interpreting Ct values from the various SARS-CoV genome detection protocols and suggests that standardization is necessary in the reporting of Ct values with respect to the target gene.

To Win the Battle, First Know Your Enemy: Error Rates in Immunohematology External Quality Assessment Results.

BACKGROUND: As some errors in pretransfusion testing remain unrecognized, error rates and the resulting need for corrective measures are probably underestimated. External quality assessment (EQA) schemes could provide valuable input for identifying error-prone laboratory tests because they are designed to monitor test performance and errors. So far, however, there are only limited published data on error rates in such schemes. METHODS: The types and incidence of incorrect results in an EQA scheme for red cell immunohematology with 187 participating laboratories were examined. The results of 58 distributions between 1999 and 2017 were evaluated, considering also the employed determination methods. RESULTS: Out of a total of 58,726 results, 563 (0.96%) were incorrect. Error rates were 5.45% for antibody identification, 1.39% for Rh phenotyping, 0.83% for serologic cross-match, 0.60% for direct antiglobulin test, 0.20% for Kell phenotyping, 0.16% for antibody screening, and 0.14% for ABO phenotyping. During the observation period, 53 participants reported error-free results, while 37 reported one incorrect result and 97 repeatedly reported incorrect results for one or more analytes. Error rates obtained by manual methods significantly surpassed those obtained by automated methods (1.04 vs. 0.42%). The introduction of double testing with two different systems reduced error rates in Rh phenotyping from 1.55 to 0.50%. CONCLUSION: Risk assessment should consider that error rates in pretransfusion test results vary. These data delineate the error risk potential of individual laboratory tests and thus should aid in tailoring appropriate improvement measures.

Evidence for the positive impact of ISO 9001 and ISO 15189 quality systems on laboratory performance - evaluation of immunohaematology external quality assessment results during 19 years in Austria.

Background ISO 9001 and ISO 15189 have been established as continuative models for quality systems beyond national laws, mandatory standards and guidelines of expert associations regarding analytical and organisational performance of medical laboratories and transfusion services. Although widely used, their impact on laboratory performance has not been investigated. Methods We retrospectively analysed the results of 167 laboratories in 59 distributions of the Austrian red cell immunohaematology external quality assessment (EQA) scheme in the years 1999-2017. The performance for each parameter and trends of individual participants were compared with respect to certification or accreditation status of participants' quality systems and to laboratory type. Results Considering more than 52,000 EQA results, the absence or presence of a laboratory quality management system showed different error rates. Laboratories with ISO 9001 or ISO 15189 certification/accreditation had 0.7% incorrect results, while this rate was doubled without such quality systems (1.4%, p=0.0002). Statistically significant error reductions were seen upon ISO 9001/ISO 15189 implementation (1.3% before vs. 0.7% after; p=0.0468). Transfusion services had fewer errors (0.9%) compared to hospital and independent laboratories (both 1.2%). Conclusions Implementation and maintenance of quality systems according to ISO 9001 or ISO 15189 as well as laboratory specialisation result in better analytical performance as can be seen in immunohaematology EQA results. The conclusion is that these results apply to other laboratory tests and perhaps to other areas of health care.

Systems medicine, personalized health and therapy.

The 7th Santorini Conference was held in Santorini, Greece, and brought together 200 participants from 40 countries in several continents, including Europe, USA but also Japan, Korea, Brazil and South Africa. The attendees had the opportunity to: listen to 60 oral presentations; participate in two lunch symposia; look at 103 posters, which were divided in two groups ('systems medicine and environment' and 'pharmacogenomics and cancer') and attend a dedicated exhibition with six companies. The meeting was organized by the Institut National de la Santé et de la Recherche Médicale (INSERM) U1122; IGE-PCV and by 'Biologie Prospective' with the collaboration of the European Society of Pharmacogenomics and Theranostics (ESPT), under the auspices of international organizations (e.g., International Federation of Clinical Chemistry and Laboratory medicine [IFCC], European Federation of Clinical Chemistry and Laboratory Medicine [EFLM], European Diagnostic Manufacturers Association [EDMA], Federation of European Pharmacological Societies [EPHAR], European Science Foundation [ESF]). The 3 days of the conference stimulated intensive discussions on systems biology and the influence of omics technologies on personalized health. Sixty speakers were invited or selected from early abstracts and gave presentations on the following topics: From systems biology to systems medicine/pharmacology; Omics/translating pharmacogenomics/proteomic biomarkers/metabolomics; Human nutrition and health/personalized medicine. We are summarizing here the main topics and presentations, according to the successive sessions.